Purdue University's research found that OpenAI's amazing chatbot, ChatGPT, answers questions on software development incorrectly more than half (50%). However, a third of participants were tricked by a bot.
The Purdue team examined ChatGPT's programming answers to 517 Stack Overflow questions to assess the accuracy, coherence, breadth, and brevity of ChatGPT's responses. In addition, US academics conducted language and sentiment analysis on the responses before asking a dozen volunteers about the model's output.
"Our analysis shows that 52% of ChatGPT answers are incorrect and 77% are verbose," the team's paper states. However, due to their thoroughness and well-articulated language style, ChatGypt answers are still favorable 39.34% of the time. 77 percent of selected ChatGPT responses require correction.
In ChatGPT, OpenAI acknowledged that its program "may provide inaccurate information about people, places, or facts." Purdue asked the lab to see if it had any comments on the study.
Who answers better? is the title of the pre-print paper. An in-depth look at ChatGPT and Stack Overflow responses to questions on software engineering. It was written by researchers David Udo-Imeh, Bonan Kou, Samia Kabir and Assistant Professor Tianyi Zhang.
According to their report, users can only spot errors in ChatGPT answers when they are obvious. Users often need to identify a solution's error or underestimate its error level, however, when the error is not easily verified or requires an external IDE or documentation.
The study found that two out of 12 people continued to choose the response as a priority, even though it contained an obvious mistake. Reportedly, ChatGPT's friendly, authoritative tone is credited for this.
According to the semi-structured interviews, the research states that "polite language, pronunciation and textbook style answers, comprehensiveness and coherence in answers make completely wrong answers seem correct".
